add memory keyword and unlimited constant for SET MEMORY LIMIT - #27
Open
puzpuzpuz wants to merge 1 commit into
Open
add memory keyword and unlimited constant for SET MEMORY LIMIT#27puzpuzpuz wants to merge 1 commit into
puzpuzpuz wants to merge 1 commit into
Conversation
javier
pushed a commit
to questdb/documentation
that referenced
this pull request
Sep 14, 2026
## Summary
Documents the new memory-limit feature across configuration, monitoring,
and access control.
**OSS — per-workload memory limits**
- `configuration/cairo-engine.md`: new "Memory limits" section covering
the reloadable workload limits (`cairo.query.memory.limit.bytes`,
`cairo.mat.view.refresh.memory.limit.bytes`,
`cairo.wal.apply.memory.limit.bytes`, plus a pointer to
`cairo.live.view.refresh.memory.limit.bytes`, which the live-views
config page already documents), per-workload breach behavior, and a note
that memory-mapped memory does not count toward a limit. Cross-links
added from the WAL, materialized-views, and capacity-planning pages.
- `query/functions/meta.md`: `query_activity` gains the `memory_used`
and `memory_limit` columns.
- `configuration/materialized-views.md`: documents
`cairo.mat.view.refresh.busy.retry.limit` and
`cairo.mat.view.refresh.busy.retry.timeout`, which govern what happens
after a refresh breaches its memory limit and were previously
undocumented. The breach description in `cairo-engine.md` links to them.
**Enterprise — per-principal memory limits**
- `query/sql/acl/alter-user.md`, `alter-service-account.md`, and a new
`alter-group.md` (+ sidebar): `SET MEMORY LIMIT { <size> | UNLIMITED }`.
- `security/rbac.md`: new "Memory limits" section — how limits resolve,
what a per-principal limit does and does not cover, how to inspect
limits, the `SET MEMORY LIMIT` permission, and `ALTER GROUP` in the
command list.
- `query/sql/show.md`: the `memory_limit` column on `SHOW USERS`, `SHOW
GROUPS` and `SHOW GROUPS <user>`, and `SHOW SERVICE ACCOUNTS` and `SHOW
SERVICE ACCOUNTS <user | group>`.
**Limit resolution — override, not min.** Limits resolve by strict
precedence: a principal's own limit → (users only) the most-restrictive
limit among its groups → the global `cairo.query.memory.limit.bytes`
workload limit. A `0`/`UNLIMITED` value at a level means "not set" and
falls through to the next. A more specific limit, when set, fully
**overrides** the broader one and **binds even when larger**, so a
per-user or per-group override can raise a principal's ceiling above the
workload limit, not only lower it. Service accounts never inherit group
limits. (Earlier drafts of these docs described a most-restrictive "min"
behavior; revised per questdb/questdb-enterprise@fb4e050 "Make
per-entity memory limits override, not min".)
## Corrections made after checking against the implementation
Two statements in earlier revisions of this PR were wrong, and both were
verified against the Enterprise code rather than re-reasoned:
- **`UPDATE` was described as never picking up a per-principal limit.**
False for a non-WAL table. `UpdateOperation.apply()` passes the caller's
own `SqlExecutionContext` to `UpdateOperatorImpl.executeUpdate`, which
calls `queryRegistry.register(...)`, which acquires a QUERY-workload
tracker from that context's security context — so a non-WAL `UPDATE`
**is** capped by `SET MEMORY LIMIT`. An operator sizing one from the old
text would have met an unexpected OOM abort. Only the WAL path (the
default table type) runs under the WAL apply job's tracker. Replaced
with a "What a per-principal limit covers" section that splits the two.
- **`SHOW GROUPS <user>` and `SHOW SERVICE ACCOUNTS <user | group>` were
shown without a `memory_limit` column**, with prose saying so
explicitly. They carry it — `AbstractShowLinkedEntitiesCursorFactory`
adds it to both metadata shapes, pinned by `ShowAclTest`. Since these
are among the results that break a positional client, the omission hid
half the breaking change.
Example tables that disagreed with each other on the same entity's limit
were also made consistent.
## Behaviour the docs previously did not mention at all
- **`COPY ... TO` exports are capped by the issuing principal's limit.**
The export runs under that principal's own context on the query
workload, so a limit must be sized for the largest single thing the
principal runs, exports included — not only for its interactive queries.
- **`SET MEMORY LIMIT` is self-escalating.** It takes no entity name, so
its holder can set *any* principal's limit including its own; combined
with override-not-min, a non-admin holding it can raise its own ceiling
above `cairo.query.memory.limit.bytes`. Documented as a warning,
together with the `GRANT ALL` expansion — a principal granted `ALL`
before the upgrade does not acquire the new permission.
- **The breaking change.** Five results gain a column on upgrade (`SHOW
USERS`, `SHOW GROUPS` and `SHOW GROUPS <user>`, `SHOW SERVICE ACCOUNTS`
and `SHOW SERVICE ACCOUNTS <user | group>`), plus `SELECT *` on
`sys.acl_entities`. Positional clients must be updated; name-based
clients are unaffected.
- **`sys.acl_entities`** stores the value, and is readable only by the
built-in admin — a principal holding `DATABASE ADMIN` is still denied.
- **`memory_limit` means different things per statement**: the
*effective* limit in `SHOW USERS`, the entity's *own* limit in `SHOW
GROUPS` / `SHOW SERVICE ACCOUNTS`, since neither inherits one.
- **An external (SSO/OIDC) user's inherited group limit refreshes at
next login**, not on its current session — the same refresh-on-login
model that already governs group-granted permissions for external users.
## Re-validation against the merged code
Both source PRs merged after the last revision of this doc PR (OSS on
2026-06-20, Enterprise on 2026-09-08), and OSS `master` moved on
afterwards. Every claim was re-checked against merged Enterprise `main`
and OSS `master` (`questdb` submodule pin `fef82615`), and these were
stale:
- **There are four workload limits, not three.** Live views
(questdb/questdb#7461) added `LIVE_VIEW_REFRESH` as a workload with its
own reloadable `cairo.live.view.refresh.memory.limit.bytes`. The
live-views config page on `main` already documents the key, so the
memory-limits section now points to it rather than duplicating it, and
the intro, capacity-planning blurb, and the RBAC "not covered by a
principal limit" list name live view refresh.
- **A mat view refresh breach no longer invalidates the view on first
breach.** questdb/questdb#7275 reclassified a refresh OOM as a
transient, retried failure: the refresh is deferred by
`cairo.mat.view.refresh.busy.retry.timeout` and the view is invalidated
only after `cairo.mat.view.refresh.busy.retry.limit` consecutive
failures (`MatViewRefreshJob.isRetriableRefreshError`, pinned by
`WorkloadMemoryTrackerTest`). A live view refresh breach, by contrast,
invalidates immediately (`LiveViewRefreshJob`). The breach paragraph is
now a per-workload list.
- **`ALTER USER ... SET MEMORY LIMIT` on an external (SSO/OIDC) user is
rejected** (`AccessListModel.setMemoryLimit`: `Cannot set memory limit
for external user`). The docs previously only said such users inherit
from groups; they now say the direct form is refused, on both `rbac.md`
and `alter-user.md`.
- **Coverage note trimmed.** The earlier "coverage is best-effort"
paragraph, with its list of tracked allocation sites, is replaced by the
one fact an operator needs: memory-mapped memory, such as the column
files a query reads, does not count toward a limit.
- **WAL apply limit rarely fires.** WAL apply runs only simple
`UPDATE`s, metadata ALTERs, and commits, so the entry now says the limit
mainly isolates WAL apply from the query budget.
- **Upgrade note.** The `memory_limit` column is added by an automatic
migration (`SysMig6`) on the first primary boot or on promote; an ACL
statement refused with a message naming the `memory_limit` column
migration means the node is not yet migrated. The window normally closes
once WAL apply catches up, so retry first; a persistent refusal calls
for a restart, which re-runs the migration.
- `query_activity`'s two memory columns are `null` for nested
registrations, which share the outer query's budget.
Confirmed unchanged and still accurate: the `SET MEMORY LIMIT` grammar
and permission name, override-not-min precedence and `0`/`UNLIMITED`
semantics (`EntMemoryTrackerProvider.effectiveLimit`), the
`memory_limit` column on all five `SHOW` results including the filtered
forms and the `enabled`/`external_alias`/`grant_option` neighbours
(`AbstractShowEntitiesCursorFactory`,
`AbstractShowLinkedEntitiesCursorFactory`, `ShowAclTest`), the `COPY ...
TO` and non-WAL `UPDATE` coverage, the breach message format
(`Unsafe.checkPerQueryAllocLimit`), reloadability of every limit
(`DynamicPropServerConfiguration`), and the built-in admin being
unlimitable.
## Dependencies
Documents:
- questdb/questdb#7184 — per-query / per-workload memory limits (OSS),
merged
- questdb/questdb-enterprise#1039 — per-principal memory limits
(Enterprise), merged
Web-console syntax highlighting for the new clause:
questdb/sql-parser#27
## Verification
`yarn build`, which this repo configures to **throw** on broken links,
broken markdown links, and broken anchors.
## Review follow-up
A level-2 `/review-pr` pass (six agents, findings verified against OSS
`fef82615` and Enterprise `main`) found no wrong technical claims. Its
major findings are fixed in 4be3ffa: stale `SHOW SERVICE ACCOUNTS`
tables on the grant and revoke assume-service-account pages, the `OUT OF
MEMORY` error tag literal and the `wal_tables()`
`errorTag`/`errorMessage` columns, the `queryId` mapping for
materialized view refreshes, `CREATE GROUP ... WITH EXTERNAL ALIAS`, the
two migration-window refusal messages, the RBAC page description, and
the `query_activity` column order with the missing `is_wal` column.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
memorykeyword and theunlimitedconstant so the QuestDB web console highlights and autocompletes theSET MEMORY LIMIT { <size> | UNLIMITED }clause.memory→keywords.ts(structural keyword, alongside the existinglimit/set)unlimited→constants.ts(value literal, alongsidenone/null)Both inserted in alphabetical order.
yarn lint,yarn build, andyarn test(6681 tests) pass locally.Context
The
SET MEMORY LIMITclause is introduced by:ALTER { USER | GROUP | SERVICE ACCOUNT } ... SET MEMORY LIMIT { <size> | UNLIMITED }Docs: questdb/documentation#462